A Skeptical Look at Screening Tests

I’m going to follow Mark Crislip’s example and recycle my presentation from The Amazing Meeting last week, not because I’m lazy or short on time (although I am both), but because I think the information is worth sharing with a larger audience.

We’ve all had screening tests and we’re all likely to have more of them, but there is a lot of misinformation and misunderstanding about what screening tests can and can’t do. Screening tests are done on populations of asymptomatic people and must be distinguished from diagnostic tests done on individual patients who have symptoms. Some tests are excellent for diagnostic purposes but are not appropriate for screening purposes.

We’re constantly being admonished to get tested for one thing or another. A typical example was a recent Dear Abby column. She got a letter from a woman who had been screened for kidney disease and learned that she had a mild decrease in kidney function. Abby was shocked to learn that 26 million Americans have chronic kidney disease, and she advised her readers to get their kidneys checked. This was terrible advice. It superficially seems like good advice, because if you have something wrong with your kidneys, you’d want to know about it, right? In fact, if there was anything wrong anywhere in your body, you’d want to know about it. By that logic, it might seem advisable to test everyone for everything. But that would be stupid. It would find lots of false positives, it would create anxiety by picking up harmless variants and anomalies that never would have caused problems, it would be expensive, and it would do more harm than good.

What makes a good screening test?

Just one example of how screening tests can harm: If we screened everyone with urine cultures, we would find asymptomatic urinary tract infections in fully 30% of elderly women. Treating those infections provides no benefit. Giving people unnecessary antibiotics would only cause side effects and contribute to antibiotic resistance in bacteria.

So we need to be selective about what tests we use for screening. Here are some of the criteria for a good screening test:

1. Disease has serious consequences
2. Screening population has high prevalence of the condition
3. Not too many false positive or false negative results
4. Test detects disease before critical point
5. Test is safe – causes little morbidity
6. Test is affordable and available
7. Treatment exists and is not too risky or toxic
8. Treatment is more effective when started earlier

Sensitivity, specificity, prevalence, and predictive values: a Bayesian analysis

Would this be a good screening test? A press release said “Breaking news! New blood test 87% accurate in detecting Alzheimer’s with an 85% specificity rate. Now we can diagnose it earlier so it can be treated effectively.”

Unfortunately there is no effective treatment for Alzheimer’s, so on that basis alone this wouldn’t be a good screening test. But let’s ignore that for the moment and look at the numbers. What does it mean that a test is 87% accurate and 85% specific?

It means the test is positive in 87% of those who have Alzheimer’s (sensitivity) and negative in 85% of those who don’t have Alzheimer’s (specificity). But those numbers are meaningless unless you know the prevalence of the disease in the population being tested. If you test a population of 10,000 people where the prevalence of Alzheimer’s is 5%, here’s what happens:

65 people with Alzheimer’s will have a false negative result and be led to believe they don’t have it when they really do. 1,425 people without Alzheimer’s will have a false positive result and be led to believe they have Alzheimer’s when they really don’t.

If you test positive, what’s the likelihood that you actually have the disease? To get that, you divide the number of true positives (435) by the total number of positives (1,860) and you get 24%. This is the positive predictive value (PPV). If you test negative, what’s the likelihood that you truly don’t have the disease? The number of true negatives (8,075) divided by the total number of negatives (8140) works out to 99%, the negative predictive value (NPV). So this would be a good test for ruling out Alzheimer’s, but it wouldn’t be good for ruling it in. A positive test would only mean there was a 24% chance that you had the disease. The predictive values will be very different for populations with a low prevalence and those with a high prevalence. Screening tests are much less accurate when the disease is rare.

Mammography

When a woman gets a mammogram, she usually gets a postcard in the mail a few days later. It will say one of two things: “Your mammogram was normal” or “Something looked suspicious; please come back for further evaluation.” If she gets the second postcard, just how worried should she be? Women who get that news tend to panic, but only 10% of them actually have breast cancer. (This percentage is average; it will be higher for women with risk factors. The numbers aren’t important; what’s important is to realize that a positive screening test is not a diagnosis. In most cases, further testing is needed.)

We tend to overestimate the benefits of mammography because of lead time bias, length bias, and over-diagnosis bias. David Gorski has explained those concepts in detail. We know mammography over-diagnoses cancers because there are 22% more diagnoses of invasive cancer in women whose cancers were detected by mammography than in unscreened women who were diagnosed with cancer. That means there must have been cases of invasive cancer in the unscreened group that spontaneously resolved or lost their invasiveness.

Evidence-based recommendations for mammography have recently changed from yearly starting at age 40 to every 2 years starting at age 50. But that’s for the general population. Women at high risk may still be advised to have earlier or more frequent tests.

Mammography saves lives, but it is not as good as most people think. If 1,000 women are screened yearly for 10 years starting at age 50:

• 2-10 will be over-diagnosed and treated unnecessarily
• 5-15 will be diagnosed earlier without any change in the final outcome
• 500 will have at least one false alarm
• 250 will undergo biopsy
• 1 life will be saved
• 999 would have lived just as long if they had never had a mammogram

(Important: these numbers are for the population as a whole. The benefits of mammography are greater for women at high risk.)

PSA testing for prostate cancer

Prostate cancer is very common but isn’t always harmful. It is found in 80% of autopsies where the men died of something else. Many more men die with prostate cancer than because of it.

The screening test for prostate cancer is a blood test for prostate-specific antigen (PSA). This is not a yes-or-no test. It must be interpreted in the context of the patient’s age and risk factors and the rate of rise, and any cut-off level is arbitrary and will miss some small percentage of cancers. If the PSA test is positive, the next step is biopsy. Typically, 12 needle biopsies are done, 6 on each side. They find cancer in 25% of patients. But if you go back and do more biopsies, you’ll find cancer in 25% more patients. Theoretically, if you could see every cell in the prostate, you might be able to find a cancer cell or two in almost everyone, most of which would never progress or kill the patient. So you have to decide how many biopsies are reasonable. If you find cancer on a biopsy, the next step is treatment, and treatments for prostate cancer are not benign.

In a large European study, screening resulted in an absolute reduction in deaths from prostate cancer of 7 per 10,000 men screened. We can look at this in terms of number needed to screen (NNS) and number needed to treat (NNT). To prevent one death from prostate cancer, 1,068 men would have to be screened and 48 treated. But here’s the kicker: there was no reduction in all-cause mortality. The overall death rate was the same in the screened group as in the unscreened group.

If a prostate cancer is localized and low grade, it is reasonable to observe the patient and not treat unless he develops signs of progression. A recent study compared surgery to watchful waiting and found no reduction in deaths. Within 30 days of surgery, 1/5 of the patients had complications including deaths. 2 years after surgery, these long term complications were present:

• 17% were incontinent
• 81% had erectile dysfunction
• 12% had bowel dysfunction

Popular advice has been “Get tested; it could save your life” but current expert advice is “Don’t get tested; it does more harm than good.” (Mainly from impotence and incontinence.) Emotional anecdotes abound. One doctor wrote an article titled “The New York Times Killed My Patient.” His patient refused PSA testing because he had read that it was not recommended; he developed invasive prostate cancer and died. Another doctor wrote about the opposite experience: his patient had insisted on testing. He was diagnosed with low-grade localized cancer, the kind that can be observed without treating. But he couldn’t face living with the knowledge that he was harboring an untreated cancer. He was afraid of surgery and opted for radiation treatment. He developed radiation proctitis and had rectal pain and bleeding for years. He became impotent and lost bladder control. He told his doctor he would rather be dead than live wearing adult diapers.

The American Urological Association initially disagreed with the recommendation not to screen, but they have re-considered and issued these new recommendations:

• Don’t screen men under 40 or over 70
• Don’t screen men with a life expectancy of less than 10-15 years
• Don’t screen men age 40-50 who are at average risk
• Consider screening men age 55-69 who are at average risk
• Consider screening high risk men of any age
• Before any screening, doctor should discuss risks and benefits with patient

The United States Preventive Services Task Force

The USPSTF is an independent group of experts who keep up with the current medical literature and issue recommendations based on the best available evidence. Their recommendations are frequently updated and available online. They’re not perfect, but they’re the most trustworthy source we have, and I rely on them. Their recommendations are not meant to be a cookbook. Their website explains:

The USPSTF recognizes that skilled clinicians serve their patients by individualizing recommendations…to the specific circumstances, values, and perspectives of the individual patient.

They recommend some screening tests be done on everyone: screening for high blood pressure and obesity. Other recommendations are different for different age, sex, or risk groups:

• Pap smear (age 21-65, every 3-5 years)
• Colorectal cancer (choice of tests, age 50-75)
• Diabetes (only if blood pressure is elevated)
• Cholesterol (men over 35; women and younger men only if they are at increased risk)

Sometimes the recommendations are complicated. Screening for osteoporosis is recommended for women aged 65 years or older and for younger women whose fracture risk is equal to or greater than that of a 65-year old white woman who has no additional risk factors. Men should not be screened. You almost need a translator to decide whether some of the recommendations apply to you: that’s what your doctor is for.

The USPSTF recommends against doing these tests as routine screening tests because they do more harm than good:

• Annual chest x-ray
• TB tine test
• Scoliosis check
• EKG
• Teaching patients to do breast and testicular self-exams

Sometimes they’re not sure, and they say so. For glaucoma, they say there is insufficient evidence to recommend either for or against screening.

A recent study found that routine physical exams don’t reduce mortality. Routine visits to a doctor can still be worthwhile, but the physical exam itself (the laying on of hands, looking in the ears, and applying a stethoscope to the chest) is largely a waste of time in the absence of symptoms.

Questionable screening tests offered to the public for profit

Several companies offer ultrasound screening. They come to town, set up in a YMCA or church, and offer “tests your doctor won’t order”: EKG, echocardiogram, ultrasounds to check for narrowing of the carotid arteries, ultrasounds of the abdominal aorta to check for aneurysms, screening for peripheral artery disease (PAD), etc. They offer glowing testimonials from people who believe they would have had a stroke or ruptured aneurysm if they had not been tested, that this screening saves lives. They don’t divulge the reason your doctor won’t order these tests: they do more harm than good. The only one the USPSTF recommends is abdominal aortic aneurysm (AAA) screening and they recommend it be done only once and only in men age 65-75 who have ever smoked. The companies will gladly test everyone and will repeat the test every year when they come back to town. Their advertising is deceptive and their testing is more likely to harm than help.

Electrodermal testing is done with a biofeedback machine hooked up to a computer. It diagnoses all kinds of bogus conditions and “imbalances” and tells the operator which homeopathic remedies and supplements to sell to the customer. It’s about as useful as a Magic Eight-Ball.

Some chiropractors use surface electromyography to locate your subluxations. The test is useless, particularly since there’s no such thing as a chiropractic subluxation.

Whole body CT scans were popular for a while; they were offered in free-standing CT facilities on request, without a doctor’s orders. They exposed patients to the same amount of radiation as 923 chest x-rays. They found abnormalities in 37-86% of patients, creating anxiety and leading to unnecessary follow-up tests that were sometimes invasive and potentially life-threatening. Fortunately they seem to have gone out of style.

When pharmacies in the UK started offering glucose and cholesterol tests, the number of visits to doctors increased, but only 10% of people with positive tests needed treatment.

Talking20 does 17 tests on a drop of blood and plans to increase this to hundreds or thousands of tests you don’t need.

Direct to consumer genetic testing is offered by several companies. They don’t test your entire genome, only specific SNPs (single nucleotide polymorphisms), and they report probabilities based on imperfect data. They might tell you that people with the same SNP as you are 30% more likely to develop Parkinson’s disease than people with other SNPs. So what? There’s nothing you can do to prevent it. And remember, genetics is not destiny. Having the gene for a disease doesn’t mean you will necessarily get the disease. Gene expression depends on environmental factors, epigenetic factors, and complex interactions with other genes. Genetic testing that promises to give you personalized diet and supplement advice is quackery. Our access to genetic information currently exceeds our understanding of what that information actually means.

The latest wrinkle is uBiome testing. It analyzes the genome of the bacteria that live on you and in you. I don’t know what you’re supposed to do with that information.

Conclusion: Screening isn’t simple

Screening tests can be very valuable but they can sometimes do more harm than good. They should be chosen judiciously, and the USPSTF offers sound recommendations based on the latest available evidence. More information is not always a good thing; sometimes it’s better not to know.